Performance modelling, analysis and prediction of Spark jobs in Hadoop cluster : a thesis by publications presented in partial fulfilment of the requirements for the degree of Doctor of Philosophy in Computer Science, School of Mathematical & Computational Sciences, Massey University, Auckland, New Zealand

Big Data frameworks have received tremendous attention from the industry and from academic research over the past decade. The advent of distributed computing frameworks such as Hadoop MapReduce and Spark are powerful frameworks that offer an efficient solution for analysing large-scale datasets running under the Hadoop cluster. Spark has been established as one of the most popular large-scale data processing engines because of its speed, low latency in-memory computation, and advanced analytics. Spark computational performance heavily depends on the selection of suitable parameters, and the configuration of these parameters is a challenging task. Although Spark has default parameters and can deploy applications without much effort, a significant drawback of default parameter selection is that it is not always the best for cluster performance. A major limitation for Spark performance prediction using existing models is that it requires either large input data or system configuration that is time-consuming. Therefore, an analytical model could be a better solution for performance prediction and for establishing appropriate job configurations. This thesis proposes two distinct parallelisation models for performance prediction: the 2D-Plate model and the Fully-Connected Node model. Both models were constructed based on serial boundaries for a certain arrangement of executors and size of the data. In order to evaluate the cluster performance, various HiBench workloads were used, and workload’s empirical data were fitted with the models for performance prediction analysis. The developed models were benchmarked with the existing models such as Amdahl’s, Gustafson, ERNEST, and machine learning. Our experimental results show that the two proposed models can quickly and accurately predict performance in terms of runtime, and they can outperform the accuracy of machine learning models when extrapolating predictions

Cardinality Enhancement of SAC-OCDMA Systems Using New Diagonal Double Weight Code

Optical code division multiple access (OCDMA) provides  another dimension to multiple access systems,  in which each user is assigned a unique code. This allows  each subscriber  to simultaneously access the medium without any contention. However, simultaneous access of multiple users introduce multiple access interference  (MAI)  which primarily deteriorates  the performance of OCDMA systems.  This paper proposes a new code called diagonal double weight (DDW) code to elevate the performance and cardinality of spectral amplitude coding  (SAC) OCDMA  systems. Performance of our proposed code is evaluated using comprehensive analytical  analysis  followed by simulation analysis. Examination  of bit error rate shows that DDW code along with  single photodiode detection  technique  provides efficient performance, with added benefits of simplified design, large cardinality and ease of implementation

Microbial β-Glucosidase: sources, production and applications

Cellulose is the most abundant biopolymer in biosphere and the major constituent of plant biomass. Cellulose polymer is made up of β-glucose units linked by β-glucosidic bonds. Cellulase is an enzymatic system that catalyzes the hydrolysis of cellulose polymer to glucose monomers. This enzymatic system consists of three individual enzymes namely endoglucanase, exoglucanase and β-glucosidase which act synergistically to degrade cellulose molecules into glucose. Cellulases are produced by bacteria, fungi, plants, and animals and used in many industrial applications such as textile industries, laundry and detergent industries, paper and pulp industry, animal feeds, and biofuels production. β-Glucosidase is a diverse group of enzymes with wide distribution in bacteria, fungi, plants and animals and has the potential to be utilized in various biotechnological processes such as biofuel production, isoflavone hydrolysis, flavor enhancement and alkyl/aryl β-D-glucoside and oligosaccharides synthesis. Thus, there is increased demand of β-glucosidase production from microbial sources under profitable industrial conditions. In this review, β-glucosidase classification, localization, and mechanism of action will be described. Subsequently, the various sources of β-glucosidase for industrial sector will be discussed. Moreover, Fermentation methods and various parameters affecting β-glucosidase production will be highlighted on the light of recent findings of different researchers. Finally, β-glucosidase applications in biofuel production, flavors enhancement, isoflavones hydrolysis, cassava detoxification and oligosaccharide synthesis will be described

Characterization of thermophilic β-Glucosidase of rhizospheric bacterial strain (LSKB15) isolated from Cholistan Desert, Pakistan

Fifty thermophilic bacterial strains isolated from rhizospheric soil of Cholistan desert, Pakistan, and designated as LSKB01-LSKB50 were screened for β-glucosidase gene (bgl) belonging to glycoside hydrolase family 1 (GH 1) using PCR technique. Subsequently, the same strains were screened for extracellular β-glucosidase production using esculin as substrate. All fifty strains were shown to be amplified for conserved region of bgl gene and to secrete extracellular β-glucosidase. One strain (LSKB15) secreted relative high amount of this enzyme as indicating by size of ferric-esculetin precipitate. This strain was further cultivated on cellulose containing media and β-glucosidase was purified by ammonium sulfate, dialysis and gel filtration chromatography. The purified enzyme showed an optimal temperature of 60°C and an optimal pH of 7. It also showed excellent temperature and pH stability retaining > 90% activity after incubation for 2 h at pH 5-8 and 40-60°C. Finally, the purified enzyme was run on Native-PAGE and subsequently incubated in phosphate buffer containing 5 mM of 4-methylumbelliferyl-β-D-glucoside (4-MUG) for 15 min at 50°C and visualized by UV light as white band. We concluded that thermophilic LSKB15 β- glucosidase may work with other cellulase to degrade available cellulose synthesized by plant and the properties exhibited by it such as high temperature and pH stability pointed out its potential industrial importance

Microbial β-Glucosidases: screening, characterization, cloning and applications

Cellulose is the most abundant biomaterial in the biosphere and the major component of plant biomass. Cellulase is an enzymatic system required for conversion of renewable cellulose biomass into free sugar for subsequent use in different applications. Cellulase system mainly consists of three individual enzymes namely: endoglucanase, exoglucanase and β-glucosidases. β-Glucosidases are ubiquitous enzymes found in all living organisms with great biological significance. β-Glucosidases have also tremendous biotechnological applications such as biofuel production, beverage industry, food industry, cassava detoxification and oligosaccharides synthesis. Microbial β-glucosidases are preferred for industrial uses because of robust activity and novel properties exhibited by them. This review aims at describing the various biochemical methods used for screening and evaluating β-glucosidases activity from microbial sources. Subsequently, it generally highlights techniques used for purification of β-glucosidases. It then elaborates various biochemical and molecular properties of this valuable enzyme such as pH and temperature optima, glucose tolerance, substrate specificity, molecular weight, and multiplicity. Furthermore, it describes molecular cloning and expression of bacterial, fungal and metagenomic β-glucosidases. Finally, it highlights the potential biotechnological applications of β-glucosidases

Redistributive Effects of Fiscal Policy across the Income Groups in the Urban-Rural Areas of Pakistan

There exist a large number of studies related to the estimates of government budgetary redistributive effects and its related problems with regard to different countries of the world.1 Studies of the impacts of government expenditures and taxes in Pakistan have been conducted within the framework of either incidence of taxes or the impact of expenditures across the income groups. The studies carried out by Azfar (1972); Jeetun (1978); Alauddin and Raza (1981) Malik and Saqib (1985, 1989) cover different aspects of taxation—tax incidence, progressivity or regressivity of the tax system across the income groups/individuals and regions. These studies did not discuss the expenditure side of the budget. Shirazi (1996) analysed the impact of government transfer programmes (Zakat and Ushr) across the income deciles. Ghaus (1989) studied the incidence of provincial and municipal government service-related expenditure benefits in Karachi metropolitan and therefore, the scope of her study was limited to one city only. Despite the existence of a rich bibliography on the subject of government redistrbutive budgetary effects and its related problems, no study is available which covers the overall redistributive impacts of government budgetary policy in Pakistan. This study explores the impacts of government expenditures and taxes on the distribution of income across various income groups along with net fiscal impacts in the urban-rural areas of Pakistan. The rest of our study is organised as follows. In the following section, Part II, we describe the methodology and data set. In Part III, the results of the study are presented. The Part IV concludes the paper

Price Clustering After the Introduction of Bitcoin Futures

Economic theory suggests that introduction of derivative contracts can improve the informational efficiency of the underlying asset prices (Danthine, 1978). In this study, we examine the impact of the introduction of Bitcoin futures on price clustering in Bitcoin. Our findings suggest that price clustering in Bitcoin meaningfully decreases post the introduction of its futures contracts

Analysis and modelling of the impact of plasma RF harmonics in semiconductor plasma processing

Reactive ion etching (RIE) has been used extensively in the last few decades in the microelectronics industry for integrated circuit fabrication. However, the monitoring and control of this process is quite challenging because the plasma process is complex and not fully understood The use of a newly developed Plasma Impedance Monitor (PIM) to monitor the RIE process is reported. The sensitivity and the usefulness of the PIM to detect the end pomt of a RIE are tested in a SF6 plasma m a capacitively coupled planar reactor with silicon (Si) and silicon dioxide (S i0 2) samples. The measurement o f harmonic components of the current-voltage (I-V) characteristics are tested as parameters to monitor the RIE processes and detect the end point when a S i 02 layer on a Si substrate undergoes SF6 RIE. The parameter used for the end pomt detection is empirically modelled as a polynomial equation of the mput factors (1 e , RF source power, chamber pressure and gas flow rate) usmg the Box-Behnken experimental design. The end point can be predicted from the modelled equation of the parameter used for the end point detection. An analytical solution for a non-symmetnc, capacitively coupled plasma driven by a non-sinusotdal radio frequency (RF) current is obtained under the assumptions of time-independent, collisionless ion motion, mertialess electrons and uniform current density. Modelling is developed considering that the RF current can be expressed as a summation of the Fourier components whose frequency is exactly an mteger multiple of the fundamental frequency. The different plasma parameters obtamed from the present model are compared with those of an established model developed by Lieberman [M A Lieberman, IEEE Trans Plasma Set, vol 16, p 638, 1988]. The present model always found the even harmonic components of IlF voltage and hence the overall RF impedance, whereas the Lieberman model found no even harmonic component of RF voltage. The sheath resistance and capacitance could always be estimated usmg the present model, whereas the Lieberman model fails to relate either of these parameters while considering the harmonic frequencies. The overall RF impedance and RF voltage obtamed from the present model are verified with experimental values. The relative magnitudes of RF voltage and impedance harmonics determined by the present model are shown to follow qualitatively the values measured m the experiment. The values of the normalized RF voltage and impedance harmonics assume lower values both for calculated and measured quantities as the asymmetry of the plasma chamber decreases

Prescribing Patterns in Hospital Inpatients

Objective: To evaluate the prescribing patterns by an audit of prescriptions in hospital inpatients. Materials and methods: After a written informed consent from the medical ward incharge & hospital administrator 32 case notes of adult patients discharged from a private hospital in Malir were collected from 1st to 30th April 2012. Patients demographics, disease & prescription details (number, type, dose, route, frequency, duration of drug use, tendency of polypharmacy, cost of drugs & discharge notes) were entered in a specially designed performa. Results: Mean age of patients was 27.18years with 14 males & 18 females. They were diagnosed to have enteric fever (10), gastroenteritis (5), RTI & others (13). Average hospital stay period was 2.5 days.Total number of drugs used were 120, of which only 5 (4.17%) were prescribed by generic name. 25.83% drugs were from National Essential drug List of Pakistan (NEDLP). Mean number of drugs per patient was 9.35. Antibiotics & analgesics each was given to 29 (90.63%) patients. Anti -ulcer drugs were given to 27(84.38%) & nebulization to 11 (34.38 %) patients without need. Average cost of drugs per patient was 1200 rupees. None of the prescription was complete for the above mentioned parameters. Conclusion:  Audit  of  prescribing  patterns  in  hospital  inpatients  of  a  private  setup  showed  irrational  use  of  drug

Association between tumour volume and recurrence of squamous cell carcinoma of the head and neck

Objective: To evaluate the prognostic significance of computerized tomography derived tumour volume for squamous cell cancers of the head and neck, treated primarily by surgery.Methods: The retrospective review study comprised 72 patients with head and neck malignancies who were treated primarily by surgery at Aga Khan University Hospital, Karachi, with/without adjuvant. It was done from May 2007 to November 2008. Each patient was followed up for a minimum of one year to check for recurrence. For statistical analysis SPSS 17 was used. Frequencies, cross-tabulations with chi square tests to find associations, binary logistic regression analysis, Cox regression analysis and receiver operating characteristic curve tests were run on the data.Results: Overall, the median tumour volume for patients with recurrent disease was 52cm3 compared to 22cm3 for those who did not have a recurrence. It was found that large tumour volume was associated with a significantly higher chance of recurrence (p = 0.009). Laryngeal cancers with volumes greater than 46cm3 and oral cancers with volumes greater than 23.1cm3 were associated with poor prognosis.CONCLUSIONS: The primary tumour volume can represent an important prognostic factor for treatment outcome. Patients with larger primary tumour volumes should be treated more aggressively